Interactive rehabilitation method and system for movement of upper and lower extremities

ABSTRACT

An interactive rehabilitation method for movement of upper and lower extremities is disclosed. An identification label of an extracted image is detected to provide an operating position of an image of an extremity. A movement mode for a target image is determined according to the identification label and the target image is displayed in a scene. It is determined whether identification labels corresponding to movement of an extremity of the target image are being continuously obtained, and, if so, the performance of the movement of the extremity is led based on operational guidance. A feedback operation is provided according to the movement of the extremity, preset movement paths and velocities, and targeted positions of the target image. It is determined whether the target image has been moved to the preset targeted positions, and, if so, the performance of the movement of the extremity is graded.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This Application claims priority of Taiwan Patent Application No.96129617, filed on 10 Aug. 2007, the entirety of which is incorporatedby reference herein.

1. Description of the Related Art

Given the aging society, it has become more apparent that many everydaycare products for older persons, do not meet or fully satisfy the needsof older persons. This is especially noticeable when looking at medicaltreatment for extremity and extremity attachments for apoplexy victims,wherein, the demand to provide more enjoyable and interesting extremityrehabilitation is growing. Note that it is assumed that currentextremity rehabilitation for older persons are extremely boring, andbecause of this in part, causes poor rehabilitation results.

One solution for the aforementioned problem, is to provide interactivegames comprising virtual computer images for apoplexy victims, which arenot only enjoyable but also provide extremity rehabilitation for theapoplexy victims. Thus, assisting to improve rehabilitation results ofapoplexy victims.

Thus, an interactive rehabilitation method and system for upper andlower extremities is desirable, assisting with extremity rehabilitationand body training (Chinese shadow boxing, for example) for apoplexyvictims via virtual computer images of interactive games.

BRIEF SUMMARY OF THE INVENTION

Interactive rehabilitation methods are provided. An exemplary embodimentof an interactive rehabilitation method comprises the following. Anidentification label of an extracted image is detected to provide anoperating position of an image of an extremity. A movement mode for atarget image is determined according to the identification label and thetarget image is displayed in a scene. It is determined whetheridentification labels corresponding to movement of an extremity of thetarget image are being continuously obtained, and, if so, theperformance of the movement of the extremity is led based on operationalguidance. A feedback operation is provided according to the movement ofthe extremity, preset movement paths and velocities and targetedpositions of the target image. It is determined whether the target imagehas been moved to the preset targeted positions, and, if so, theperformance of the movement of the extremity is graded.

Interactive rehabilitation systems are provided. An exemplary embodimentof an interactive rehabilitation system comprises a hand positionmonitoring module, a target image movement control module, an imagefeedback module, and a movement evaluation module. The hand positionmonitoring module detects an identification label of an extracted imageto provide an operating position of an image of an extremity. The targetimage movement control module determines a movement mode for a targetimage according to the identification label, displays the target imagein a scene, determines whether identification labels corresponding tomovement of an extremity of the target image are being continuouslyobtained, and, if the identification labels are being continuouslyobtained, leading the movement of the extremity based on operationalguidance. The image feedback module provides a feedback operationaccording to the movement of the extremity, preset movement paths andvelocities and targeted positions of the target image. The movementevaluation module grades the movement of the extremity when the targetimage has been moved to the preset targeted positions.

A detailed description is given in the following embodiments withreference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention can be more fully understood by reading the subsequentdetailed description and examples with references made to theaccompanying drawings, wherein:

FIG. 1 is a schematic view of a computer system of the presentinvention;

FIG. 2 is a schematic view of an interactive rehabilitation system 100shown in FIG. 1 of the present invention;

FIG. 3 is a flowchart of an interactive rehabilitation method of thepresent invention;

FIG. 4 illustrates human extremities;

FIG. 5 illustrates a behavioral range of the operator detected by animage extraction device;

FIG. 6 illustrates grabbing a sphere in a game scene;

FIG. 7 illustrates feedback states in response to operator movements inthe game scene; and

FIGS. 8-11 illustrate Chinese shadow boxing motions.

DETAILED DESCRIPTION OF THE INVENTION

Several exemplary embodiments of the invention are described withreference to FIGS. 1 through 3, which generally relate to interactiverehabilitation for movement of upper and lower extremities. It is to beunderstood that the following disclosure provides various differentembodiments as examples for implementing different features of theinvention. Specific examples of components and arrangements aredescribed in the following to simplify the present disclosure. Theseare, of course, merely examples and are not intended to be limiting. Inaddition, the present disclosure may repeat reference numerals and/orletters in the various examples. This repetition is for the purpose ofsimplicity and clarity and does not in itself dictate a relationshipbetween the various described embodiments and/or configurations.

The invention discloses an interactive rehabilitation method and systemfor mobility of upper and lower extremities, assisting extremityrehabilitation and body training (Chinese shadow boxing, for example)for apoplexy patients via virtual computer images of interactive games.

Given the aging society, the invention provides an interactive game forolder persons to play the game indoors, and provide brain stimulus andentertainment to facilitate independence of older persons. Additionally,the video system of the game enables older persons to play with andinteract with their children or other players, which assists in socialinteraction, thus, slowing the aging process in mind and soul.

An embodiment of an interactive rehabilitation method and system forextremities can serve as training equipment for interactive extremityrehabilitation, immediately leading operators to perform extremityrehabilitation or training exercise via the game.

FIG. 1 is a schematic view of a computer system of the presentinvention.

An embodiment of an interactive extremity rehabilitation system 110 isimplemented in a computer device 130. The computer device 130 is wiredor wireless-connected to an image extraction device (a Webcam, forexample) 150. The image extraction device 150 can be internallyinstalled in the computer device 130. The interactive extremityrehabilitation system 110 extracts real-time images of a person via theimage extraction device 150 and transmits the extracted images to thecomputer device 130 to be displayed in a user interface (not shown)provided by the interactive extremity rehabilitation system 110.Additionally, the image extraction device 150 comprises an imageidentification system program for analyzing an image scope of a reactionarea, retrieving movements from the start to the end of extremities, andperforming real-time operations for dynamic images and returningfeedbacks for flexibility training of extremities.

FIG. 2 is a schematic view of an interactive extremity rehabilitationsystem 110 shown in FIG. 1 of the present invention. FIG. 3 is aflowchart of an interactive rehabilitation method of the presentinvention.

The exemplary embodiment of an interactive rehabilitation system 110comprises a hand position monitoring module 210, a target image movementcontrol module 230, an image feedback module 250, and a movementevaluation module 270. A process for the exemplary embodiment of theinteractive rehabilitation system 110 is first described in thefollowing.

Referring to FIGS. 1-3, a color mark or recognizable mark (defined as anidentification label in this embodiment) for a position is first placedon a detected portion of an operator for extraction by the imageextraction device 150 before the rehabilitation process starts. When agame provided by the present invention is activated, the interactiverehabilitation system 110 detects extremity movements (hand movements,for example) of the operator using the image extraction device 150. Thehand position monitoring module 210 detects the identification label ofan extracted image of the operator extracted by the image extractiondevice 150 to provide a corresponding position (defined as an extremityposition) for extremities in a game scene (step S31).

The target image movement control module 230 retrieves theidentification label corresponding to the movement of the extremitiesfrom the hand position monitoring module 210 to determine movement modesand appearance sequences of a target image (step S32). The systempredefines required target images and classifications (Chinese shadowboxing motions or sphere grabbing actions, for example, which are not tobe limitative). Each targets image and classification comprises pluralmovements and movement paths and velocities, and targeted positions arepreset to movements of each target image. The preset data is stored in adatabase (not shown). Extremity movements of the operator correspond tomovements of the target image. When a movement of an extremitycorresponding to the identification label is retrieved, the target imagemovement control module 230 immediately selects a movement mode and anappearance sequence of the target image corresponding to the movement ofthe extremity and displays the target image (step S33), a Chinese shadowboxing motion or a sphere grabbing action, for example.

The target image movement control module 230 determines whetheridentification labels corresponding to movement of an extremity of thetarget image are being continuously retrieved from the hand positionmonitoring module 210 (step S34), i.e. determining whether the operatorperforms the Chinese shadow boxing motion or sphere grabbing action. Ifthe identification labels are not continuously retrieved, whichindicates that the operator did not completely perform the movement,does not know how to perform the movement, or has forgotten how toperform the movement, the operator is reminded how to perform themovement by arrow guidance or other eye-catching suggestions. If theidentification labels are being continuously retrieved, the extremities(the hands, for example) of the operator is led based on the presetmovement paths and velocities and the targeted positions via operationalguidance (step S35). The operator, for example, is led to grab a targetimage in a game scene and place the target image at a correct targetposition or perform a Chinese shadow boxing motion.

The image feedback module 250 provides a feedback operation for theoperator according to the movement of the extremity, the preset movementpaths and velocities and the targeted positions (step S36). Shapes,emotional expressions and sounds of the target image, for example, arechanged or an error message (image) or sound effect is shown. When thehand image (i.e. the movement of the extremity) of the operator overlapsthe target image (while grabbing the target image) or velocity or locusdifference (fast and slow motions of the Chinese shadow boxing motions)therebetween is generated, the feedback operation is provided. Thefeedback operation indicates image pattern variation and combinations ofsound and power outputs or velocity variation, enabling the operator toexperience interactions with the target image.

The target image movement control module 230 determines whether all ofthe target images have been moved to the preset targeted positions (stepS37). That is, when a game for grabbing spheres is performed, whethereach sphere is placed at its individual position, or, when a game forChinese shadow boxing motions is performed, whether all Chinese shadowboxing motions are completed, is determined. If a target image has notbeen moved to the preset targeted positions, the process proceeds tostep S33 to repeat the described operations and enable all of the targetimages to be moved to their preset targeted positions. When all of thetarget images have been moved to their preset targeted positions, themovement evaluation module 270 grades the movement of the extremities ofthe operator according to similarity between the movement of theextremities and the target images (step S38), and then the processterminates.

As described, the interactive extremity rehabilitation system 110enables patients requiring rehabilitation for hand extremities toimplement movement training via game interactions. Additionally, thesystem can provide competition for more than one user at the same gameplatform via video conference, achieving enjoyable rehabilitation andrequired training results.

Processes for components of the interactive extremity rehabilitationsystem 110 are described as follows.

The hand position monitoring module 210 performs skin color recognition(based on the mark placed on the extremities) using computer visionsimulation and tracks dynamic object behaviors according to recognitionresults. Further, the hand position monitoring module 210 extractsimages from real-time images retrieved from the image extraction device150 according to preset skin color definitions, determines whether eachpixel on the extracted image comprises an area identical to that of thepreset skin color definitions, marking a center of the area, subtracts aposition of the center from that of a center of an actual screen, andtransmits a control signal of a resulting distance vector to the targetimage movement control module 230 for tracking.

With respect to the target image movement control module 230, acomputer/computer game system provides target images for different typesof games and movements, movement paths and velocities, targetedpositions, and parameters are preset to each target image. The movementpaths and velocities, targeted positions, and parameters are definedaccording to medical treatment requirements. The target image movementcontrol module 230 leads, controls, and corrects hand movement of theoperator to grab and place the target image to a correct targetposition, correcting and rehabilitating hand function of patients.

Processes for the target image movement control module 230 are describedas follows.

To achieve dragging of a target image (a game object) via gestures, anoperational scope for a gesture operating area is first locked and anavailable skin color is separately highlighted using skin colordetection. The dragged gesture represents a dynamic process, the gestureoperating area provides dynamic signals of a frame, and an availabledynamic signal of the gesture operating area is extracted using a framedifferential detection method. Next, a logical operation (AND, forexample) is implemented to the skin color area with the dynamic signalsof the dragged gesture to generate a skin color differential area (i.e.the area in which the gestures of a frame are performed). The skin colordifferential area corresponds to coordinate positions in the game space,collisions for the skin color differential area and a movement area ofthe target image are detected, and collision signals serve asdetermination for selecting a game object. Additionally, to facilitatethe target image change, coordinate positions based on the draggedgesture, and the average center coordinate data of the skin colordifferential area is corresponded to coordinate positions in the gamespace to generate target coordinates used for leading the target imageto move.

With respect to the image feedback module 250, when the hand image ofthe operator overlaps (i.e. the grabbing movement) the target image orvelocity or locus difference between the hand movement and the targetimage movement is generated, the image feedback module 250 provides afeedback operation with image pattern variation and combinations ofsound and power outputs or velocity variation for the operator based onpreset parameters. The image feedback module 250 provides leading,controlling, and correcting the hands of the operator to grab and placethe game object to a correct target position according to preset values.

Processes for the image feedback module 250 are described as follows.

Movement paths and velocities of a target image are created andparameters of targeted positions of the target image are defined and thedefined data is stored in a database (not shown). Additionally, it isdetermined whether movement values, generated by operational behaviorusing artificial intelligence (IA), correspond to system definedstandard parameters. When the hand images of the operator overlaps (i.e.the grabbing movement) the target image or movement velocities or lociof the hand image and the target image in the game scene are differentfrom the system predefined values (i.e. the predefined parameters), afeedback operation with image pattern variation and combinations ofsound and power outputs or velocity variation is provided for theoperator based on preset information stored in the database.

With respect to the movement evaluation module 270, a real-time feedbackmode is available to the operator according to the movement paths andvelocities and targeted positions, so that the operator can beimmediately corrected.

Several examples are described to illustrate the process of theinteractive extremity rehabilitation system 110.

Referring to FIG. 4, human extremities can be at least classified as awrist swinging around (as shown in Fig. A), a lateral movement (as shownin Fig. B), a finger winding movement (as shown in Fig. C), andclenching movements (as shown in Figs. D-F). Sphere grabbing motions orChinese shadow boxing actions can be implemented using the describedmovements.

For clenching movements, human-machine interactions and imagerecognition design are applied to achieve accuracy of movement operationand correctness, as the system of the invention provides feedbackoperations for each movement of the operator. Image pattern variationand combinations of sound and power outputs or velocity variation, forexample, enables the operator to experience interaction with the targetimage. The movement evaluation module 270 determines performance gradesaccording to the interaction between the hand image and the targetimage.

FIG. 5 illustrates a behavioral range of the operator detected by theimage extraction device 150. The extractible range (ER) of the imageextraction device 150 is shown by the block, wherein the extremities(Ex.) of the operator can only perform inside of the block and will notbe detected outside of the block.

Referring to sphere grabbing in FIG. 6, when the game starts, the systemselects and sets an identification label for tracking the operator anddisplays a target image corresponding to a selected movement mode. Thesystem detects and displays extremities of the operator in a game scene,wherein when the operator grabs a sphere (the target image) in the gamescene, the system leads the operator to place the grabbed sphere at acorrect target position according to preset targeted positions andparameters stored in a database (not shown), and provides feedbackaccording to velocity or locus similarity of the movement of theextremity. When the current sphere is placed to a correct targetposition and feedback is provided, the system then displays anothersphere in the game scene and leads the operator to place the sphere to acorrect target position.

The system leads grabbing movements of the operator according to presetmovement paths and velocities for each target image and, when the handimage of the operator overlaps (i.e. the grabbing movement) the targetimage or velocity or locus difference between the hand movement and thetarget image movement is generated, leads and corrects movements of theoperator based on image movements, emotional expressions, or movingdirections. Additionally, the system provides a feedback pattern(located at any position on the sphere or the operating window) to showfeedback states in response to operator movements in the game scene.Referring to FIG. 7, FIG. A illustrates a normal state where the spherehas not been grabbed, FIG. B illustrates touching the sphere by theextremity image of the operator, and FIG. C illustrates interactionsbetween the extremity image and the sphere, such that the operator cansynchronously experience interactions from the target image during theextremity rehabilitation process.

Referring to Chinese shadow boxing motions in FIGS. 8-11, when the gamestarts, the system selects and sets an identification label for trackingthe operator and displays a target image corresponding to a selectedmovement mode. The system detects and displays extremities of theoperator in a game scene, wherein when the operator motions, the systemdetermines movements of the operator as Chinese shadow boxing motions,leads the extremities (both hands in this embodiment) of the operator tomove to a correct target position with a correct path using a virtualfigure, and provides feedback according to velocity or locus similarityof the movement of the extremities.

Referring to FIG. 8, the system generates and locates a virtual figure(VF) at the left side of the frame and enables the left hand (LH) andthe right hand (RH) of the virtual figure to perform correspondingmovements according to preset targeted positions and parameters storedin the database (not shown), facilitating the operator to imitate themovements of the virtual figure. The right side of the frame shows areal figure extracted by an image extraction device. When the operatorswings both hands, the real figure in the frame generates correspondingmovements. The system determines whether a movement of the operator iscorrect based on the movement of the real figure and that of the virtualfigure and provides feedback (performance grading, for example). Whenthe movement is complete, the system shows another virtual figure of thenext Chinese shadow boxing motion (as shown in FIG. 9) and leads theoperator to imitate the motion. The described process is repeated toenable the operator to complete the subsequent Chinese shadow boxingmotions (as shown in FIGS. 10 and 11) and feedback (performance grades,for example) is provided based on the completed motions, such that theoperator can correct his movements according to the feedback.

An embodiment of the interactive rehabilitation method and systempromotes flexibility of older persons and improves the extremity abilityof the operator via real extremity tanning. Additionally, the inventionprovides human-machine interactions to improve degeneration of extremityability for older persons caused by old age via a physical touchplatform. The extremity activities for older persons are thus expandedand the game platform allows enjoyable entertainment and recreationalactivities which improve reaction degeneration of older persons. Thatis, extremity mobility of patients are improved and influencedunobtrusively and imperceptibly by playing games.

Methods and systems of the present disclosure, or certain aspects orportions of embodiments thereof, may take the form of a program code(i.e., instructions) embodied in media, such as floppy diskettes,CD-ROMS, hard drives, firmware, or any other machine-readable storagemedium, wherein, when the program code is loaded into and executed by amachine, such as a computer, the machine becomes an apparatus forpracticing embodiments of the disclosure. The methods and apparatus ofthe present disclosure may also be embodied in the form of a programcode transmitted over some transmission medium, such as electricalwiring or cabling, through fiber optics, or via any other form oftransmission, wherein, when the program code is received and loaded intoand executed by a machine, such as a computer, the machine becomes anapparatus for practicing and embodiment of the disclosure. Whenimplemented on a general-purpose processor, the program code combineswith the processor to provide a unique apparatus that operatesanalogously to specific logic circuits.

While the invention has been described by way of example and in terms ofthe preferred embodiments, it is to be understood that the invention isnot limited to the disclosed embodiments. To the contrary, it isintended to cover various modifications and similar arrangements (aswould be apparent to those skilled in the art). Therefore, the scope ofthe appended claims should be accorded the broadest interpretation so asto encompass all such modifications and similar arrangements.

1. An interactive rehabilitation method, comprising: providing path characteristics of a target image; extracting a movement of an extremity from an extraction device; enabling the movement of the extremity to interact with the target image in a scene; and immediately adjusting interaction states between the movement of the extremity and the target image according to the path characteristics.
 2. The interactive rehabilitation method as claimed in claim 1, further comprising providing a feedback operation according to the interaction states between the movement of the extremity and the target image.
 3. The interactive rehabilitation method as claimed in claim 1, further comprising leading the movement of the extremity based on operational guidance according to preset parameters corresponding to the target image to lead the movement of the extremity to interact with the target image and providing a score according to interactive similarity.
 4. An interactive rehabilitation method, comprising: detecting an identification label of an extracted image to provide an operating position of an image of an extremity; determining a movement mode for a target image according to the identification label; displaying the target image in a scene; determining whether identification labels corresponding to movement of the extremity of the target image are being continuously obtained; if the identification labels are being continuously obtained, leading the movement of the extremity based on operational guidance; providing a feedback operation according to the movement of the extremity, preset movement paths and velocities and targeted positions of the target image; determining whether the target image has been moved to the preset targeted positions; and the target image has been moved to the preset targeted positions, grading the performance of the movement of the extremity.
 5. The interactive rehabilitation method as claimed in claim 4, further comprising, when plural target images are provided, determining movement modes of the target images and appearance sequences of each target image according to the identification label.
 6. The interactive rehabilitation method as claimed in claim 4, further comprising, if the identification labels are being continuously obtained, leading the movement of the extremity based on the operational guidance, the preset movement paths and velocities and the targeted positions of the target image corresponding to the target image.
 7. The interactive rehabilitation method as claimed in claim 4, further comprising grading the performance of the movement of the extremity according to similarity between the movement of the extremity and the target image.
 8. An interactive rehabilitation system, comprising: a hand position monitoring module, detecting an identification label of an extracted image to provide an operating position of an image of an extremity; a target image movement control module, determining a movement mode for a target image according to the identification label, displaying the target image in a scene, determining whether identification labels corresponding to movement of an extremity of the target image are being continuously obtained, and, if the identification labels are being continuously obtained, leading the movement of the extremity based on operational guidance; an image feedback module, providing a feedback operation according to the movement of the extremity, preset movement paths and velocities and targeted positions of the target image; and a movement evaluation module, when the target image has been moved to the preset targeted positions, grading the performance of the movement of the extremity.
 9. The interactive rehabilitation method as claimed in claim 8, wherein, when plural target images are provided, the target image movement control module determines movement modes of the target images and appearance sequences of each target image according to the identification label.
 10. The interactive rehabilitation method as claimed in claim 8, wherein, if the identification labels are being continuously obtained, the target image movement control module leads the movement of the extremity based on the operational guidance, the preset movement paths and velocities and targeted positions of the target image corresponding to the target image.
 11. The interactive rehabilitation method as claimed in claim 8, wherein the movement evaluation module grades the movement of the extremity according to similarity between the movement of the extremity and the target image.
 12. The interactive rehabilitation system as claimed in claim 9, wherein the target image movement control module separately highlights an available skin color area for the movement of the extremity using a skin color detection method.
 13. The interactive rehabilitation system as claimed in claim 8, wherein the target image movement control module extracts available dynamic signals for the movement of the extremity using a frame differential detection method.
 14. The interactive rehabilitation system as claimed in claim 8, wherein the target image movement control module implements a logical operation to the parameters of the available skin color area with the dynamic signals to generate a skin color differential area.
 15. The interactive rehabilitation system as claimed in claim 8, wherein the image feedback module provides the feedback operation according to the preset movement paths and velocities and the targeted positions when the movement of the extremity overlaps the target image or velocity or locus difference therebetween is generated.
 16. A computer-readable storage medium storing a computer program providing an interactive rehabilitation method, comprising using a computer to perform the steps of: detecting an identification label of an extracted image to provide an operating position of an image of an extremity; determining a movement mode for a target image according to the identification label; displaying the target image in a scene; determining whether identification labels corresponding to movement of an extremity of the target image are being continuously obtained; if the identification labels are being continuously obtained, leading the movement of the extremity based on operational guidance; providing a feedback operation according to the movement of the extremity, preset movement paths and velocities and targeted positions of the target image; determining whether the target image has been moved to the preset targeted positions; and if the target image has been moved to the preset targeted positions, grading the performance of the movement of the extremity.
 17. The computer-readable storage medium as claimed in claim 16, further comprising, when plural target images are provided, determining movement modes of the target images and appearance sequences of each target image according to the identification label.
 18. The computer-readable storage medium as claimed in claim 16, further comprising, if the identification labels are being continuously obtained, leading the movement of the extremity based on the operational guidance, the preset movement paths and velocities and the targeted positions of the target image corresponding to the target image.
 19. The computer-readable storage medium as claimed in claim 16, further comprising grading the performance of the movement of the extremity according to similarity between the movement of the extremity and the target image. 